What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory

We develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit activations data. propose new function space, which is reminiscent classical bounded variation-type spaces, that captures compositional structure associated networks. derive representer theorem showing ReLU are solutions regularized data problems over from this space. The space consists compositions Banach spaces second-order variation in Radon domain. These sparsity-promoting norms, giving insight into role sparsity network have skip connections and rank weight matrices, providing theoretical support for these common architectural choices. problem we study can be recast as finite-dimensional training regularization schemes related notions decay path-norm regularization. Finally, our analysis builds on techniques spline theory, between splines.